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Abstract 


Recent  research  efforts  strive  to  address  the  growing  need  for  dismount  surveillance, 
dismount  tracking  and  characterization.  Current  work  in  this  area  utilizes  hyperspectral 
and  multispectral  imaging  systems  to  exploit  spectral  properties  in  order  to  detect  areas  of 
exposed  skin  and  clothing  characteristics.  Because  of  the  large  bandwidth  and  high 
resolution,  hyperspectral  imaging  systems  pose  great  ability  to  characterize  and  detect 
dismounts.  The  acquisition  and  modification  of  hyperspectral  images  presents  several 
challenges  and  inconveniences  for  researchers.  They  are  dependent  on  a  particular 
environmental  condition.  They  are  costly,  cumbersome,  and  involve  dedicated  software 
practices  and  particular  knowledge  to  operate.  A  simple  system  where  the  development 
and  enhancement  of  multi-data  dismount  models  becomes  a  grand  necessity.  This  thesis 
demonstrates  a  computer  aided  multi-data  fused  dismount  model,  for  greater  ease  of 
dismount  detection,  characterization  and  identification.  The  system  is  created  by  fusing: 
pixel  mapping,  signature  attachment,  and  pixel  mixing  algorithms. 

The  developed  multi-data  dismount  model  produces  simulated  hyperspectral  images 
that  closely  represents  an  image  collected  by  a  hyperspectral  imager.  The  dismount  model 
can  be  modified  to  fit  the  researcher’s  needs.  The  multi-data  model  structure  allows  the 
employment  of  a  database  of  signatures  acquired  from  several  sources  (i.e., 
Spectroradiometer,  or  hyperspectral  imagers).  The  model  is  flexible  enough  to  allow 
further  exploitation,  enhancement  and  manipulation.  For  example,  the  addition  of  facial 
details,  addition  of  background  objects,  dismount  skin  signatures,  clothing  signatures, 
modification  to  the  position,  pose,  size,  and  appearance  of  the  dismount  and  more.  The 
multi-data  dismount  model  developed  in  this  effort  fulfills  the  need  for  a  dismount 
modeling  tool  in  a  hyperspectral  imaging  environment. 
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COMPUTER  AIDED  MULTI-DATA  FUSION  DISMOUNT  MODELING 


1  Introduction 

United  States  Armed  Forces  and  Law  Enforcement  communities  continue  to  stress 
the  importance  of  research  into  persistent  surveillance  capabilities  that  provide  detection 
and  characterization  of  dismounts.  The  United  States  Air  Force  (USAF)  in  Doctrine 
Document  1  (AFDD-l)  states  that  “As  a  leader  in  military  application  of  air,  space,  and 
intelligence,  surveillance,  and  reconnaissance  technology,  the  Air  Force  is  committed  to 
innovation  to  guide  research,  development,  and  fielding  of  unsurpassed  capabilities”  [3]. 

Due  to  the  general  shift  from  conventional  warfare  to  terrorism  and  urban  warfare, 
locating  and  tracking  individuals  of  interest  has  become  of  vital  importance  [3].  Recent 
research  efforts  address  this  growing  need  for  dismount  surveillance,  as  well  as  dismount 
tracking  and  characterization.  These  research  efforts  include, 

1.  Defense  Advance  Research  Project  Agency  (DARPA)  Combat  Zones  that  See 
program  (CTS)  [2],  which  has  the  goal  of  creating  networks  of  cameras  and  sensors 
to  monitor  everything  that  moves  and  report  observations  to  an  operation  center. 

2.  United  States  Army  research  effort  towards  the  development  of  algorithms  for  the 
Unmanned  Air  Vehicle  (UAV)  ISR  system,  with  the  objective  of  tracking  targets  in 
urban  and  others  environments  [6] . 

The  conception  of  a  model  to  simulate  dismounts,  as  well  as  the  scene  background,  is 
necessary.  Therefore  the  thesis,  combine  several  dismount  characteristics,  clothing  and 
environmental  signatures,  dismount  detection  and  characterization,  in  a  multi-data 
modeling  environment  to  create  a  dismount  model. 
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1.1  Problem  Statement 


Recent  research  efforts  [12,  16,  29,  15]  have  provided  the  ability  to  detect  dismounts 
(humans)  from  the  background  in  different  environment  conditions.  In  [12],  a  robust 
method  for  dismount  detection  is  proposed,  based  on  skin  detection  using  near-infrared 
wavelengths.  Several  different  techniques  previously  studied  on  dismount  characterization 
and  detection  include,  hyperspectral  skin  detection  [12,  16],  blinking  detection,  movement 
detection,  face  detection  and  recognition  [37],  and  red-green-blue  (RGB)  skin 
characterization. 

The  goal  of  this  thesis  is  to  create  and  demonstrate  the  feasibility  of  a  systematic 
method  to  construct  a  multi-data  fused  dismount  model  for  the  demanding  studies  of 
detection,  characterization  and  identification.  The  techniques  of  our  dismount 
representation  include  current  studies  in  skin  detection,  face  detection,  and  skin  and 
clothing  characterization.  The  integration  of  these  approaches  are  used  to  render  an 
dismount  model.  Using  the  multi-data  dismount  model,  an  operator  or  an  algorithm  will 
recognize,  characterize,  and  identify  distinctive  features  of  specific  individuals  for  use  in 
classification  and  characterization.  Furthermore,  the  system  should  provide  a  platform 
capable  of  supplementary  expansion  for  the  later  integration  of  new  capabilities  for 
dismount  modeling. 

1.2  Scope 

Limitations  of  this  project  include  the  availability  of  hyperspectral  images  containing 
several  individuals  of  different  ethnicity,  pose  positions  and  clothing  arrangements.  In 
order  to  achieve  our  objective,  the  tasks  accomplished  by  this  research  effort  are  the 
following: 

1.  Explore  methods  for  dismount  representation  and  data-fused  modeling. 
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2.  Analyze  image  acquisition  technologies  used  to  detect  dismounts  and  skin 
(hyperspectral,  multispectral  and  Red-Green-Blue). 

3.  Research  and  study  different  dismount  detection  techniques  and  their  capabilities 
with  the  objective  of  meeting  information  requirements  for  such  techniques. 

4.  Research  the  approaches  towards  dismount  characterization  and  identification. 

5.  Develop  a  fusion  based  computer  aided  multi-data  dismount  model. 

6.  Acquire  hyperspectral  signatures,  i.e.  skin  tones,  backgrounds,  dismount 
shape/contours,  and  clothing. 

7.  Integrate  the  previous  tasks  to  create  a  modeled  scene. 

8.  Integrate  sub-pixel  mixing  for  final  scene  rendering. 

The  result  is  a  computer  aided  dismount  model  that  provides  the  basis  for  a  complete 
and  accurate  dismount  modeling  research  tool.  The  use  of  the  system  can  grow  the 
potential  of  generating  improvements  on  characterization  of  individuals  by  an  efficient, 
easy  to  acquire,  quick  to  develop,  robust  human  modeling  design. 

1.3  Document  Organization 

Chapter  II  provides  a  discussion  of  research  literature  related  to  human  modeling. 
This  chapter  also  contains  the  background  information  on  several  approaches  for  human 
characterization,  different  types  of  multi-data  signatures  to  be  used  in  the  model,  pixel 
mixing  and  finally  a  basic  edge  detection  algorithm. 

Chapter  III  presents  the  methodology  by  describing  the  systematic  creation  of  the 
data-fused  dismount  model.  It  describes  the  architecture  of  the  system  and  explains  in 
details  the  steps  for  the  creation  of  the  multi-data  dismount  model.  This  chapter  also 
provides  details  of  the  possible  configurations  for  the  dismount  and  scene  model. 
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The  experimental  results  and  analysis  are  provided  in  chapter  IV.  This  includes  the 
validation  of  functionality  and  design  of  the  dismount  model,  integrating  several 
hyperspectral  signatures  from  different  sources.  Chapter  IV  also  compares  the 
resemblance  of  the  computer  aided  dismount  model  versus  a  hyperspectral  picture  from  a 
camera  and  gives  advantages  and  disadvantages  of  the  proposed  dismount  model. 

Chapter  V  summarizes  the  results,  and  discusses  the  impact  of  this  thesis  on 
dismount  modeling  towards  the  studies  of  characterization  and  identification,  and  provides 
recommendations  for  future  work  in  enhancing  dismount  modeling. 
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2  Background 


An  overview  of  some  prominent  approaches  for  dismount  modeling,  as  well  as,  the 
necessary  background  information  on  imaging  spectroscopy  for  multi-data  fusion  and  its 
function  for  sub-pixel  mixing  is  discussed  in  this  chapter. 

Section  2.1  introduces  several  approaches  for  dismount  modeling.  These  include 
morphological  models  and  topological  models.  Section  2.3  introduces  imaging 
spectroscopy  and  the  approaches  towards  dismount  and  scene  representation  in  an  image. 
Finally,  Section  2.4  describes  methods  for  sub-pixel  mixing  with  the  objective  of 
mimicking  realistic  dismount  representation. 

2.1  Dismount  Modeling 

There  are  two  primary  dismount  representations  are:  physiology,  and  behavioral  [9]. 
The  physiological  representation  refers  to  shape  or  geometry  of  the  dismount  [33].  A 
behavioral  representation  is  related  to  gait  (manner  of  walking),  voice  sounds,  or 
individual  proceedings  [9].  For  the  purpose  of  this  thesis,  the  focus  is  on  the  human 
ontology  representation  that  specifically  encompasses  the  visual  physiology  of  a  dismount. 

Ontology  is  a  formal  representation  of  knowledge  as  a  set  of  concepts  within  a 
domain  and  the  relationship  between  those  concepts  [38].  An  ontological  representation 
of  a  dismount  is  closely  linked  to  the  graphical  representation.  It  is  necessary  to  switch 
from  the  graphical  representation  to  the  ontology  representation  and  vice  versa.  This  is 
possible  due  to  the  use  of  human  shape  descriptors  like  landmarks,  body  composition, 
body  segmentation,  skeletonisation,  body  representation  using  geometrical  shapes,  and  the 
relation  between  them. 

Abstract  forms  of  dismount  modeling  (computer  aided  dismount  modeling)  are 
necessary  to  study  dismount  ontological  representations.  These  concepts  are  based  on 
human  morphological  and  topological  models. 
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2.1.1  Morphological  Models.  Morphology  is  a  branch  of  bioscience  dealing  with 
the  study  of  the  form  and  structure  of  organisms  and  their  specific  structural  features  [19]. 
This  includes  aspects  of  the  outward  appearance  (shape,  structure,  color,  pattern). 

The  analysis  of  the  dismount  shape  aims  to  extract  features  such  as  body  landmarks, 
skeletal  structure  and  body  composition.  These  features  serve  for  human  representation 
and  modeling  purposes. 

2. 1.1.1  Body  Landmarks  Mediods.  In  the  Fig.  2.1  and  Fig.  2.2  it  is 
demonstrated  how  the  author  of  [39]  applied  fuzzy  logic  to  extract  landmark  features 
(armhole,  neck,  chest,  crotch  and  belly  button)  from  three  dimensional  (3D)  body  scans. 
This  approach  uses  landmark  extraction  based  on  an  analysis  of  the  human  shape. 


Figure  2.1:  Human  body  morphological  landmarks  [39]. 


There  are  three  recognized  categories  of  dismounts  body  landmarks  [30]: 

1.  Locally  defined  landmarks,  defined  in  terms  of  structures  close  to  a  point;  for 
example,  an  intersection  between  three  structures. 

2.  The  intermediate  landmarks,  includes  points  such  as  the  edge  of  the  structure,  or 
local  minima  and  maxima  of  a  curvature.  They  are  defined  in  terms  of  local 
features,  but  they  are  not  surrounded  on  all  sides  like  the  first  type. 
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Figure  2.2:  Samples  of  landmark  feature  extraction  from  body  scans  (arm  hole,  neck,  chest, 
crotch  and  belly  button)  [39]. 

3.  Points  far  away  from  the  landmark,  which  are  commonly  defined  in  terms  of  a  point 
“furthest  away”  from  another  point  is  the  last  landmark. 

In  addition  to  landmarks,  there  are  “semi-landmarks”,  which  are  points  along  a  curve. 
Their  position  along  the  curve  is  arbitrary  but  these  points  provide  information  about  the 
curvature  in  multiple  dimensions. 

In  order  to  compare  different  dismount  models,  using  landmark  extraction,  shape 
analysis  must  be  accomplished.  By  definition,  shape  is  not  altered  by  translation,  scale  or 
rotation  [25].  Therefore,  to  compare  shapes,  the  non-shape  information  is  removed  from 
the  landmark  coordinates.  In  one  step,  the  shapes  are  translated  (the  same  two  coordinates 
are  fixed  to  those  values),  scaled  (to  unit  baseline  length)  and  rotated  as  explained  in  [10]. 

An  alternative,  and  preferred  method,  is  procrustes  superimposition.  This  method 
translates  the  center  of  the  shapes  to  (0,  0).  The  ’x’  center  of  the  centroid  is  the  average  of 
the  x-coordinates  of  the  landmarks  of  an  individual.  The  ’y’  coordinate  of  the  centroid  is 
the  average  of  the  y-coordinates.  Shapes  are  scaled  to  unit  centroid  size,  which  is  the 
square  root  of  the  summed  squared  distances  of  each  landmark  to  the  centroid.  The 
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configuration  is  rotated  to  minimize  the  deviation  between  it  and  a  reference,  typically  the 
mean  shape.  In  the  case  of  semi-landmarks,  variation  in  position  along  the  curve  is  also 
removed.  For  the  case  of  curved  semi-landmarks  shape  space,  analysis  is  accomplished  by 
projecting  shapes  onto  a  space  tangent  to  the  shape  space.  Within  the  tangent  space, 
conventional  multivariate  statistical  methods,  such  as  multivariate  analysis  of  variance  and 
multivariate  regression,  can  be  used  to  test  statistical  hypotheses  about  the  shape. 

Researches  have  discovered  that  this  method  of  procrustes-based  analysis  has  some 
limitations  as  mentioned  in  [31,  38]. 

1.  The  procrustes  superimposition  uses  a  least-squares  criterion  to  find  the  optimal 
rotation.  This  variation  is  localized  to  a  single  landmark  and  transferred  out.  This  is 
called  the  “Pinocchio  Effect.” 

2.  The  superimposition  may  impose  a  pattern  of  covariation  on  the  landmarks. 

3.  Features  that  are  not  captured  by  landmarks  and  semi-landmarks  cannot  be 
analyzed.  Moreover,  there  are  criticisms  of  procrustes-based  methods  that  motivate 
an  alternative  approach  to  analyzing  landmark  data. 

It  is  mentioned  in  [37]  that  the  computational  methods  involved  in  the  extraction  of 
features,  such  as  landmarks  or  shape  segmentation,  must  comply  with  two  main 
constraints: 

1 .  Landmark  extraction  and  morphological  segmentation  results  must  be 
anthropometrically  consistent. 

2.  Landmark  extraction  and  morphological  segmentation  results  must  be  consistent 
and  almost  invariant  from  one  data  set  to  another. 

The  method  of  landmark  based-modeling  may  be  useful  for  dismount  representation. 
Although,  based  on  the  constraints  described  above,  it  is  concluded  that  the  use  of 


8 


dismount  landmark  features  with  the  purpose  of  characterization,  may  present  a  challenge 
in  regards  to  data  acquisition  and  computational  analysis.  Also,  when  considering  the  field 
of  computer  vision,  it  has  been  demonstrated  that  this  approach  is  highly  dependent  on  the 
landmark  feature  detection  algorithm.  A  minimum  discrepancy  in  correctly  identifying  a 
landmark  position  size  or  shape,  or  a  landmark  parse,  will  present  a  problem  for  an 
accurate  representation. 

2. 1.1.2  Body  Outline  Method.  According  to  [8],  pertinent  information  about 
an  object  is  often  contained  in  the  shape  of  its  boundaries.  Experiments  have 
demonstrated  that  human  vision  is  capable  of  recognizing  an  object  by  the  shape  of  its 
boundaries  alone.  For  example,  Fig.  2.3  shows  a  human  body  outline.  This  recognition 
approach  is  implemented  in  [8]  using  the  Hough  Transform. 

Outline  analysis  is  another  approach  to  analyzing  shape.  Outline  analysis  is 
distinguished  from  other  approaches  by  the  use  of  coefficients  of  mathematical  functions 
that  are  fitted  to  points  sampled  along  the  outline.  Older  techniques  such  as  the  “fit  to  a 
polynomial  curve”  [22]  and  principal  components  quantitative  analysis  [39],  have  been 
superseded  by  the  eigen-shape  analysis  [19],  and  elliptical  Fourier  analysis  [25] 
approaches.  Eigen-shape  analysis  [28]  involves  fitting  a  preset  number  of  semi-landmarks 
at  equal  intervals  around  the  outline  of  a  shape.  This  is  shown  in  [10]  by  recording  the 
deviation  of  each  angle  from  one  semi-landmark  to  another  semi-landmark  of  a  simple 
circle.  Elliptical  Fourier  analysis  defines  the  outline  as  the  sum  of  the  minimum  number  of 
ellipses  required  to  mimic  the  shape  [31]. 

Body  outline  methods  have  their  weaknesses;  the  most  dangerous  (and  easily 
overcome)  is  their  susceptibility  to  noise  in  the  outline  [38].  Fikewise,  neither  compares 
homologous  points,  and  global  change  carries  more  weight  than  local  variation  (which 
may  have  large  consequences  in  regards  to  human  representation). 
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Figure  2.3:  Representative  outline  of  the  human  body,  it  is  used  to  calculate  and  analyze 
human  body  shape  [7]. 

Eigen-shape  analysis  [28]  requires  an  equivalent  starting  point  to  be  set  for  each 
specimen,  which  can  be  a  source  of  error.  Elliptical  Fourier  analysis  [25]  also  suffers  from 
redundancy  in  that  not  all  variables  are  independent  [38].  On  the  other  hand,  it  is  possible 
to  apply  both  techniques  to  complex  curves  without  having  to  define  a  centroid.  This 
makes  removing  the  effect  of  location,  size  and  rotation  much  simpler  as  explained  in  [38]. 

The  perceived  failings  of  the  body  outline  method  is  that  points  of  a  homologous 
origin  are  not  compared,  and  it  oversimplifies  complex  shapes  by  considering  only  the 
outline  and  not  internal  changes.  Ballard  [8]  identified  that  the  approximation  of  the  body 
outline  as  a  shortcoming,  hence  it  deals  poorly  with  pointed  shapes. 

Another  criticism  of  the  body  outline-based  methods  is  that  they  ignore  homology.  A 
famous  example  of  this  problem  is  that  it  mistakes  a  scapula  for  a  potato  chip  [24].  Such  a 
comparison  would  not  be  possible  if  the  data  were  restricted  to  biologically  homologous 
points. 


2. 1.1. 3  Skeletal  Essence  and  Body  Composition.  Anatomical  features,  i.e. 
skeletal  structure,  is  the  basis  of  another  type  of  dismount  representation.  The  skeleton  of 
a  body  is  the  locus  of  the  center  of  all  its  interior  maximal  circles  (2D)  or  spheres  (3D) 
[14]  (Fig.  2.4).  Together  with  the  associated  radius  function,  which  is  the  radius  of  the 
maximal  ball  around  any  given  point  on  the  skeleton,  the  object  can  be  represented  with 
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less  information  than  the  object  itself.  Such  a  technique  can  be  applied  in  human  shape 
representation  (or  description).  Commonly,  skeletal  representations  can  be  derived  by: 

1.  Computing  a  distance  transform  that  yields  the  shortest  distance  from  each  interior 
point  of  a  region  to  its  border  or, 

2.  By  identifying  at  each  point  the  largest  possible  size  of  the  primitive  shape  such  that 
it  is  entirely  contained  in  a  region  to  be  represented. 

The  skeletal  representation  is  derived  by  identifying  the  primitives  having  locally 
maximal  sizes.  These  approaches  are  straightforward,  but  computationally  costly  since  the 
distance  computation  must  be  performed  at  every  point. 

The  authors  in  [14]  present  a  concept  of  obtaining  the  skeleton  of  3-D  polyhedral 
regions  in  a  computationally  efficient  manner.  The  efficiency  of  the  approach  results  from 
the  use  of  an  intermediate,  analog  representation  of  the  given  shape  information,  which  is 
called  the  potential  field.  Potential  field  representation  avoids  the  expensive  task  of 
computing  the  distance  transform  at  each  pixel.  The  computation  is  limited  to  the 
locations  of  the  locally  maximal  primitives. 

The  computation  of  the  potential  field  can  be  performed  efficiently  if  the  shape 
information  is  given  in  a  compact  form,  i.e.  the  polyhedral  representation  of  the  surface. 
This  compact  description  may  be  directly  available  as  a  part  of  the  specification  of  the 
given  object  shape  or  it  may  be  derived  from  the  given  object  data  such  as  its  surface  or 
volume  descriptions. 

The  international  standard  anatomical  position  is  the  position  that  provides  a 
reference  point  for  describing  the  individual  structures  of  the  human  body.  In  this 
position,  the  body  is  standing  erect  with  good  posture  and  the  face  is  looking  directly 
forward  in  a  normal  manner.  The  feet  are  close  together,  flat  on  the  floor  and  the  toes 
pointing  forward.  The  arms  are  down  at  the  sides  with  the  palms  turned  forward.  It  is 
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Figure  2.4:  Polyhedral  representation  using  skeletal  essence  and  body  composition  [4]. 


necessary  for  all  of  the  individuals  to  be  in  this  position  in  order  to  correctly  approximate 
skeletal  structure  and  body  composition  [29].  Therefore,  to  effectively  use  this  approach 
as  comparison  the  position  needs  to  be  standard  and  universal  between  every  sample. 

In  conclusion,  the  use  of  anatomical  characteristics  for  human  representation  presents 
several  constraints  as  mentioned  above.  The  position,  location  and  points  of  reference 
need  to  be  standard  between  all  of  the  samples.  Computational  limitations  may  take  place 
(depending  on  the  technique)  and  the  method  is  highly  dependent  on  edge  detection 
algorithms. 

2.1.2  Topological  Model.  Topology-based  modeling  approach  utilize  3-D  point 
clouds  [29]  or  topological  structures  of  3-D  models  [24]  as  shown  in  Fig.  2.5  to  measure 
the  similarity  between  different  human  bodies  or  objects. 

The  difficulties  of  the  topology-based  approach  include  automatic  topology 
extraction  from  all  types  of  3-D  models,  and  the  discrimination  between  topologies  from 
different  individuals.  The  authors  in  [24]  propose  a  novel  method  called  Topology 
Matching  where  the  similarity  between  polyhedral  models  are  quickly,  accurately,  and 
automatically  calculated  by  comparing  Multiresolutional  Reeb  Graphs  (MRG). 

The  MRG  [24]  represents  the  topological  structure  of  a  3-D  shape  at  various  levels  of 
resolution.  The  MRG  is  created  using  a  continuous  function  on  the  3-D  shape,  which  may 
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preferably  be  a  function  of  geodesic  distance.  This  function  is  invariant  to  rotation  or 
translation  and  is  also  good  against  variations  in  connectivity  caused  by  a  mesh 
simplification  or  subdivision.  The  similarity  between  3-D  shapes  is  calculated  using  a 
coarse-to-fine  strategy  while  preserving  the  consistency  of  the  graph  structures,  which 
results  in  establishing  a  correspondence  between  the  parts  of  objects.  According  to  the 
author,  the  similarity  calculations  of  this  approach  is  fast  and  efficient.  It  is  not  necessary 
to  determine  the  particular  pose  of  a  3-D  shape,  such  as  a  rotation,  in  advance. 

Experiments  conducted  in  [24]  indicate  that  Topology  Matching  provides  a  fast  and 
efficient  computation  of  the  similarity  and  correspondence  between  shapes  and  provides 
results  that  agree  well  with  human  intuition. 


Figure  2.5:  Human  topological  representation  [41]. 

One  advantage  for  a  topology-based  approach  in  regards  to  dismount  representation 
is  that  it  leads  to  highly  similar  characteristics  between  two  identical  3-D  models  with 
different  gestures  or  poses,  which  is  not  able  to  be  effectively  accomplished  using 
shape-based  methods.  However,  shape-based  approaches  result  in  a  high  similarity 
between  3-D  models  with  different  connections  among  parts,  whereas  a  topology-based 
approach  cannot  do  the  same.  This  is  illustrated  by  considering  that  fingers  and  a  shoulder 
of  a  dismount  model  are  both  parts  of  a  human  body.  However,  the  topologies  are  quite 
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different  whether  the  finger  does  or  does  not  connect  to  a  human  body,  yet  finger  shapes 
are  simiiar  [24] . 

2.2  Digital  Imaging  and  Remote  Sensing  Image  Generation 

The  Digital  Imaging  and  Remote  Sensing  Image  Generation  (DIRSIG)  model  is  a 
synthetic  image  generation  model  engineered  by  the  Digital  Imaging  and  Remote  Sensing 
Laboratory  at  Rochester  Institute  of  Technology  [36].  The  model  can  produce 
multispectral  or  hyperspectral  imagery  from  the  visible  through  the  thermal  infrared 
region  of  the  electromagnetic  spectrum,  which  is  the  same  capability  of  the  proposed 
model  in  this  thesis. 

The  model  developed  by  DIRSIG,  as  well  as  the  thesis  proposed  model,  can  be  used 
to  test  image  system  designs,  to  create  test  imagery  for  evaluating  image  exploitation 
algorithms  and  for  creating  data  for  training  image  analysts.  The  approach  for  this  thesis 
differs  from  DIRSIG  in  basic  concept.  This  thesis  focuses  on  modeling  dismounts  in  a 
scene,  where  DIRSIG  focuses  on  scene  modeling  and  currently  does  not  incorporate 
dismounts. 

2.3  Imaging  Spectroscopy 

Many  different  types  of  image  data  can  be  used  to  represent  dismounts.  The  most 
common  systems  employs  visible  range  parameters,  near-infrared,  far  infrared,  or  some 
combination  of  these  (Fig.  2.6). 

2.3.1  Visual  Spectra  Images.  Common  standard  color  cameras  take  images  at 
three  different  wavelengths  of  light,  corresponding  to  red,  green  and  blue  (RGB).  The 
visible  region  of  the  electromagnetic  spectrum  is  frequently  used  to  depict  dismounts  due 
to  the  accessibility  and  low  acquisition  costs.  Monochrome  (gray-scale)  and  RGB 
cameras  are  commonly  used  in  this  matter.  For  example,  dismount  representation  using 
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Figure  2.6:  Electromagnetic  Spectrum.  Hyperspectral  images  use  near-infrared,  far- 
infrared  and  visible  segments  of  the  Electro  Magnetic  Spectrum  [1], 


RGB  and  gray-scale  can  be  found  in  dismount  detection  [32,  18],  and  face  recognition 
[37],  It  has  been  found  that  these  approaches  are  effective  in  limited  conditions  and  further 
development  is  necessary  for  a  robust  dismount  detection,  recognition  and 
characterization. 

2.3.2  Far- Infrared  Images.  An  alternative  to  the  visible  region  dismount 
representation  is  to  acquire  an  image  with  the  far-infrared  spectrum.  This  technique  has 
several  prominent  benefits  for  the  remote  sensing  community  and  dismount 
characterization  [20].  This  type  of  remote  sensing  is  strongly  characterized  by  thermal 
emissions.  Consequently,  dismounts  appear  brighter  than  the  background  and  tend  to 
stand  out  well.  Additionally,  image  factors  such  as  lighting,  color,  texture,  and  shadowing 
that  would  normally  disrupt  exposure  in  visual  light  images  have  less  of  an  impact  in  the 
infrared  range. 

However  infrared  imagery  does  have  its  own  limitations.  Many  objects  including 
animals,  vehicles,  electric  boxes,  roads,  light  poles,  etc.,  produce  additional  “bright  areas” 
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in  infrared  images.  This  is  especially  true  during  the  summer  time.  Also,  the  image 
intensities  of  the  same  object  are  not  uniform  and  are  adversely  affected  by  dismount 
orientation,  clothing  and  accessories.  There  is  less  image  definition  when  the  temperature 
of  the  background  environment  is  similar  that  of  the  target.  The  intensity  range  of  infrared 
images  is  generally  more  limited  and  consequently  offer  lesser  quality  data  as  compared  to 
the  visible  range  [16]. 

2.3.3  Near  and  Short-wave  Infrared.  Near  infrared  (NI)  and  short-wave  infrared 
(SWIR)  imagery  present  another  option  for  dismount  representation.  While  they  do  not 
offer  the  benefits  of  detecting  thermal  emissions,  they  have  been  proven  useful  in  skin 
detection  (Fig.  2.7)  and  face  detection  applications. 


TOSOnrn  image  1 580  nm  image  RGB  image 


Figure  2.7:  Original  imagery  from  the  1080nm,  1580nm,  and  RGB  cameras  used  for  skin 
detection  [16]. 


2.3.4  Hyperspectral  Imagery.  Hyperspectral  images  are  typically  used  for 
geological  surveys,  because  they  offer  richness  of  spectral  data  which  provides  several 
advantages  over  the  other  methods.  Hyperspectral  cameras  take  an  image  using  several 
hundred  wavelengths,  ranging  from  visible  though  the  near  infrared,  typically  400  - 
2500nm  with  an  approximated  bandwidth  of  10 nm  for  each  channel  [34].  The  light  source 
curve  (most  of  the  time  the  Sun)  and  the  absorption  features  of  the  atmosphere  dominate 
the  general  shape  of  the  spectrum  from  a  hyperspectral  system.  The  Sun’s  light  peaks  in 
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the  green  wavelengths  and  decreases  towards  longer  and  shorter  wavelengths.  The 
atmosphere  absorbs  light  at  wavelengths  that  correspond  to  the  absorption  characteristics 
of  the  constituent  gases,  such  as  nitrogen,  oxygen,  carbon  dioxide,  and  water  vapor.  The 
peaks  and  valleys  of  a  spectrum  not  due  to  the  Sun  or  the  atmosphere  reveal  information 
about  the  chemical  composition  of  the  spectrum  being  examined.  Since  every  material  has 
a  unique  spectrum,  an  analyst  can  look  for  diagnostic  features  from  hyperspectral  data 
[11].  This  additional  information  allows  analysts  to  examine  characteristics  and  identify  a 
possible  nature  of  the  imaged  materials,  as  seen  in  Fig.  2.8.  A  standard  hyper-spectral 
image  can  be  thought  of  as  a  cube,  where  the  X-Y  plane  is  the  two  dimensional  image, 
and  the  Z  plane  are  the  wavelengths  divided  in  bandwidth  sections. 


400 


2500 


wavelength,  nm 


Figure  2.8:  Representation  of  a  hyper- spectral  image.  Each  spatial  element  has  a 
continuous  spectrum  that  is  used  to  analyze  the  material  composition  [11]. 

Hyperspectral  imagery  contains  a  lot  of  information  about  the  material  being  imaged. 
Different  materials  respond  differently  at  various  wavelengths.  Analyzing  a  single  pixel 
throughout  a  hyper- spectral  image  provides  important  information  regarding  the  material 
content  in  the  image  at  that  pixel.  These  signals  can  be  used  to  discriminate  between 
multiple  classes  of  materials  in  images  as  demonstrated  in  [15]  for  clothing  material,  [12] 
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skin  detection  and  for  ground  materials.  One  shortcoming  of  hyper- spectral  sensing  is  that 
the  spatial  resolution  is  often  worse  than  panchromatic  sensors.  This  trade-off  between 
spectral  and  spatial  resolution  arises  in  the  sensor  design  as  a  result  of  the  need  to  maintain 
imaging  sensitivity  at  the  finer  spectral  resolution  of  the  hyper- spectral  sensor  [26]. 

2.4  Pixel  Mixing 

In  order  to  create  a  robust  dismount  model  that  accurately  represents  common  image 
acquisition,  an  understanding  of  “mixed”  pixels  is  necessary.  A  mixed  pixel  is  when  one 
pixel  contains  several  different  features  or  classes  as  shown  in  2.9.  This  leads  to  the 
spectral  response  at  a  pixel  being  a  mixture  of  the  underlying  classes. 


Figure  2.9:  Mixed  Pixels  at  class  edge  boundary  between  skin  and  brick  background  end- 
members  (some  shirt  pixels  are  also  mixed). 

Most  of  the  current  studies  in  the  area  of  remote  sensing  research  have  proposed 
several  techniques  for  the  un-mixing  of  pixels.  There  is  not  much  information  on  the 
mixing  of  pixels.  The  purpose  of  this  thesis  is  to  generate  a  dismount  model,  therefore  the 
use  of  techniques  for  un-mixing  pixels  are  applied  in  order  to  mix  pixels.  There  are  two 
major  approaches  to  mixed  pixel  classification:  the  linear  and  the  non-linear  approach. 
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2.4.1  Linear  Mixture  Model.  The  simplest  and  most  used  approach  is  a  linear 
mixture  model  (LMM)  [5].  This  approach  models  the  linear  combination  of  boundary 
members  or  end-members.  The  linear  mixing  model  relies  on  two  major  assumptions, 

1 .  The  spectrum  of  a  mixed  pixel  is  a  linear  combination  of  the  spectra  of  the 
constituent  pure  classes. 

2.  The  spectral  proportions  of  the  pure  classes  reflect  the  area  class  proportions. 

Linear  Mixture  Model  (LMM)  is  defined  as: 

Xi  =  Mf  +  e  (2.1) 

where,  M  is  an  N  xC  matrix,  its  columns  represent  the  spectra  of  the  different 
end-members  present  in  the  image.  xt  =  (xi,x2, ...,  xN)T  are  the  image  values  in  the  ith 
band,  f  =  [/j,  f2, ...,  fc\r  is  the  proportion  of  each  pixel  within  each  class  type  C.  N 
represents  the  number  of  features,  C  is  the  number  of  classes,  and  e  is  the  error  or  bias. 
LMM  has  two  major  constraints  in  regards  to  the  proportion  values  f. 


0  <  fi  <  1  and 


/i  +  h  +  —h  -  1 


2.4.2  Non-Linear  Methods.  The  other  pixel  mixing  approach  is  the  non-linear 
model.  This  approach  utilizes  several  statistical  techniques  for  pixel  mixing,  i.e. 
Nearest-Neighbor,  Bayes  and  Kernel  methods  [23]. 

2.5  Summary 

This  chapter  outlines  several  dismount  representation  approaches  and  their 
application  towards  dismount  identification  and  characterization.  The  chapter  also 
describes  various  alternatives  of  imagery  data  which  allows  researchers  to  accomplish 
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dismount  representation  and  characterization.  Finally,  the  LMM  algorithm  for  mixing 
pixels  is  discussed.  The  details  and  the  methods  implemented  for  a  computer  aided 
multi-data  dismount  model  are  presented  and  described  in  Chapter  III. 
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3  Methodology 


The  methodology  for  the  conception  of  a  computer  aided  multi-data  dismount  model 
is  discussed  in  this  chapter.  The  dismount  model  description  is  explained  in  detail.  Future 
developers  and  users  will  be  able  to  modify  and  expand  this  structured  design.  The 
chapter  also  discusses  the  methods,  processes  and  algorithms  used  to  design  a  modeled 
scene.  The  first  topic  is  the  architecture  and  top  level  design.  The  top  level  design  allows 
understanding  on  how  the  dismount  model  was  developed  and  the  various  modules  of  the 
model  are  individually  described.  These  modules  include, Pixel  Map,  Background  Model 
(Nature,  Man  Made  Elements),  Dismount  Model  (Face,  Feet,  Torso,  Hand,  Legs,  Neck 
and  Arms),  Signature  Attachment  (at  each  element  in  the  pixel  matrix),  End  Member 
Detector  and  Pixel  Mixing  (LMM). 

3.1  Top  Level  Design 

Figure  3.1  shows  the  procedures  for  the  development  of  a  modeled  scene.  The  first 
step  is  to  create  the  pixel  map.  The  pixel  map  is  used  to  describe  each  pixel  according  to 
its  location  in  a  modeled  scene,  as  explained  in  Section  3.2.  Once  the  pixel  map  is 
completed,  each  pixel,  or  group  of  pixels  of  a  class  (i.e.  shirt,  hair,  etc.)  is  assigned  a 
signature.  The  type  of  multi-data  signature  to  be  attached  to  the  pixel  map  depends  on  the 

end  user  modeling  needs  and  the  modeled  imager  capability.  These  multi-data  signatures 
can  be  RGB,  multi-spectral,  hyper-spectral,  imager  specific  or  any  other  required 
bandwidth. 

After  the  signatures  are  applied  to  the  map  in  its  entirety,  pixel  mixing  by  way  of 
linear  mixing  process  is  accomplished.  Pixel  mixing  creates  a  representative  model  of 
current  imagine  technologies.  To  implement  pixel  mixing  the  boundaries  of  each  modeled 
element  needs  to  be  identified.  For  this  purpose,  end-member  finder  (edge  detector) 
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Figure  3.1:  Multi-data  fusion  dismount  model  elements. 


algorithm  is  employed.  After  signatures  are  attached  and  end-member  pixel  elements  are 
mixed,  the  multi-data  dismount  model  is  complete. 

3.2  Pixel  Mapping 

In  order  to  develop  a  modeled  scene  the  creation  of  a  pixel  map  is  necessary.  The 
pixel  map  consist  of  an  m  x  n  matrix.  Therefore,  if  the  user  needs  an  image  that  is  800  by 
600  pixels,  the  dimensions  of  the  pixel  matrix  can  be  adjusted  to  render  that  image 
(m  =  800  and  n  =  600).  The  x  axis  corresponds  to  the  horizontal  dimension  and  the  y  axis 
corresponds  to  the  vertical  dimension  as  shown  in  Fig.  3.2. 

This  pixel  mapping  matrix  allows  for  the  representation  of  several  distinctive  classes 
in  the  image.  The  value  of  each  element  in  the  matrix  represents  a  class  (i.e.,  brick,  grass, 
skin,  shirt,  etc.)  in  the  scene.  For  example,  in  Fig.  3.2  a  basic  pixel  map  is  shown.  For  this 
example,  the  matrix  is  a  25  x  25  element  matrix  with  several  classes  represented. 
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Figure  3.2:  Pixel  Map  of  a  25  by  25  image.  Each  entry  in  the  matrix  represent  a  class  in 
the  scene.  For  example,  the  pixel  in  location  (1,1)  represents  background  (coded  as  0)  and 
pixel  in  location  (12,  6)  represents  skin  (coded  as  1)  as  seen  in  Table  3.2. 


For  each  pixel  map  (matrix),  a  legend ,  which  describes  each  mapped  class,  must  be 
included.  Table  3.2  shows  the  legend  for  the  pixel  map  that  belongs  to  the  pixel  map  in 
Fig.  3.2.  The  legend  describes  the  class  of  each  pixel  in  the  scene  and  identifies  which 
signature  is  being  modeled  in  the  pixel  map.  For  example,  the  pixel  in  location  (1,1) 
coded  as  0  represents  a  background  class  and  pixel  in  location  (12,  6)  represents  a  skin 
class  pixel,  coded  as  1.  Based  on  this  legend  and  the  pixel  map,  signatures  can  be  attached 
as  desired  by  the  user. 

3.2.1  Pixel  Map  Development.  There  are  several  alternatives  for  the  development 
of  a  pixel  map, 

•  Manual  Model  Development  (MMD) 

•  Modifying  an  Existing  Model  (MEM) 
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Table  3.1:  Pixel  Map  legend  codes  for  Fig.  3.2. 


Elements 

Maps  to: 

0 

Background 

1 

Skin  (Face/Head) 

2 

Skin  (Neck) 

3 

Clothing  (Torso) 

4 

Skin  (Arms) 

5 

Clothing  (Lower  Torso) 

6 

Clothing  (Legs) 

7 

Skin  (Hands) 

•  Algorithmic  Model  Generation  (AMG) 

Manual  Model  Development  (MMD)  refers  to  the  creation  of  a  two-dimensional 
(2D)  matrix  (i.e.  m  x  n),  where  each  entry  represents  a  pixel  in  the  image  according  to  its 
location  (x,  y)  in  the  pixel  matrix.  MMD  can  be  created  using  several  computer  programs 
that  allow  manipulation  of  2D  matrices,  like  Microsoft  Excel  or  Libre  Office. 

The  second  alternative  is  to  modify  an  already  created  model  (MEM).  The  model  can 
be  modified  for  several  reasons.  One  is  to  incorporate  new  variations  or  details,  and 
second,  to  add  new  desired  elements  (i.e.  additional  dismounts,  cars,  trees,  etc.).  This  is 
possible  by  changing  the  pixel  map.  For  example,  if  a  change  to  the  pixel  map  in  Fig.  3.2 
is  needed,  it  can  be  easily  achieved  by  changing  elements  of  the  matrix  with  a  different 
number  according  to  the  legend  (Table  3.2)  or  by  adding  extra  elements  to  the  pixel  map 
and  to  the  legend. 

The  third  option  is  algorithmic  modeling  or  matrix  manipulation  (AGM).  Algorithmic 
Modeling  refers  to  the  creation  of  a  pixel  map  by  the  use  of  algorithms.  Predefined 
modeled  structures  can  be  easily  modified,  adjusted  and  created  with  algorithms  and 
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clustering  schemes.  There  exist  several  programs  (i.e.  MATLAB,  Java,  Phyton)  that  allow 
clustering  identification,  matrix  management,  manipulation  and  programming. 

3.3  Model  Overview 

The  development  of  the  model  consist  of  several  stages.  The  first  stage,  is  the  “basic” 
stage.  The  most  basic  scene,  for  the  purpose  of  this  thesis,  must  include  the  dismount  and 
background  models.  Later,  on  further  stages,  more  elements  and  details  like  for  example: 
extra  dismounts,  face  details,  cars,  buildings,  trees  could  be  added  to  the  scene  to 
represent  a  different  situation  as  desired  by  the  user.  Another  alternative  to  further 
enhance  the  model  is  to  move  the  angle  of  view  in  the  scene  (sidewise  view,  top  view, 
back  view),  to  create  shadowing  effects  or  to  completely  remove  the  dismount(s)  to  model 
an  aerial  to  ground  photograph  or  a  geological  survey. 

3.3.1  Dismount  Module.  A  morphological  model  using  body  outline  and  body 
composition  by  parts  is  implemented  in  order  to  create  the  dismount  model.  The  dismount 
elements  to  be  modeled  consist  of  the  following:  head  or  face,  neck,  torso,  hands,  arms, 
legs  and  feet. 

The  combination  of  all  these  create  a  complete  dismount.  By  modeling  each 
dismount  part  separately,  we  can  accelerate  the  assignment  of  a  specific  signature  to  a 
dismount  section.  During  the  second  stage,  details  and  variations  can  be  incorporated  to 
each  section  separately.  This  enhances  the  creation,  modification  and  reuse  of  models.  For 
example,  if  the  user  desires  the  hands  to  be  of  a  specific  skin  color  or  size,  the  requested 
color  signature  could  be  assigned  to  the  entire  hands  section  instead  of  going  pixel  by 
pixel.  Once  a  dismount  is  completed  it  could  be  used  in  different  scenes  with  minimum 
changes. 

3.3.2  Background  Module.  Every  modeled  scene  needs  a  background.  For  the 
basic  stage,  the  background  model  is  a  plain  black  background  (Fig.  3.3).  Note  that  this 
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“basic”  background  can  later  be  changed  with  any  wanted  signature.  The  signature  could 
be  of:  RGB,  multispectral  or  hyperspectral  nature  as  needed  by  the  user  (Fig.  3.1). 
Modifying  elements  in  the  background  section  will  make  possible  the  simulation  of 
different  environments  and  background  objects.  Alterations  to  the  background  are  done  by 
changing  the  pixel  map  and  adding  the  respective  signatures  to  be  attached  to  the  model. 

3.4  Signature  Attachment 

Once  the  pixel  mapping  is  complete,  signatures  can  be  attached  according  to  the  pixel 
map’s  legend.  The  pixel  map  legend  allows  the  creation  of  a  rendered  scene  from  a 
repository  of  signatures,  which  can  be  assigned  to  each  element  of  the  pixel  map. 


Figure  3.3:  Basic  scene  with  RGB  signature  attached. 


For  example,  in  Fig.  3.3  the  following  RGB  colors  are  assigned  to  the  pixel  map: 
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1.  [63  72  204]  a  variation  of  blue  to  the  legs  to  represent  jeans, 

2.  [255  218  230]  to  represent  skin  in  head  and  hands, 

3.  [163  73  164]  to  represent  a  purple  shirt, 

4.  [127  127  127]  for  gray  shoes, 

5.  [0  0  0]  for  the  black  background. 

If  the  user  needs  a  different  signature,  (i.e.,  a  hyperspectral  signature),  it  is  attached 
using  the  same  scheme.  The  only  difference  is  that  a  hyperspectral  signature  has  more 
elements  per  pixel  location  instead  of  three  as  does  the  RGB  vector  [red,  blue,  green]  (Fig. 


Figure  3.4:  Pixel  map  showing  RGB  signature  (left)  vs  hyper- spectral  signature  (right). 
The  hyper-spectral  signature  have  more  elements  than  a  RGB  which  has  only  three. 


For  the  purposes  of  this  thesis,  the  signatures  will  be  gathered  with  an 
spectroradiometer  (ASD)  [21]  and  from  several  hyperspectral  images  acquired  with  a 
hyperspectral  camera  (HST)  [35].  Signatures  can  also  be  acquired  using  previously 
created  models  or  from  a  signature  repository.  This  depends  on  the  requirements  for  the 
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model  (i.e.,  if  the  model  is  of  hyperspectral  nature  then  the  signatures  needs  to  be 
hyperspectral).  Environmental  effects  also  need  to  be  considered  for  the  selection  of  the 
appropriate  repository  of  signatures.  Discussions  of  the  different  considerations  when 
selecting  a  hyperspectral  signature  are  mentioned  in  Section  4.3. 

3.4.1  Signatures  from  Spectroradiometer.  The  spectroradiometer  used  for  the 
acquisition  of  hyper-spectral  signatures  is  a  FieldSpec  3  Portable  Spectroradiometer  from 
ASD  Inc.  [21].  The  spectroradiometer  consists  of  a  hand-held  contact  probe  connected  to 
a  spectrometer  with  a  fiber  optic  cable  (Fig.  3.5).  The  contact  probe  contains  its  own 
illumination  source  (artificial)  which  covers  the  Visual  (VIS),  Near  Infrared  (NIR)  and 
Short-Wave  Infrared  (SWIR).  The  system  is  calibrated  by  placing  the  contact  probe,  with 
its  illumination  source,  against  a  calibrated  reflectance  panel  with  a  known  reflectance. 
Once  the  system  is  calibrated,  the  contact  probe  is  placed  against  materials  such  as 
dismount’s  skin  and  clothing.  Reflectance  for  the  different  wavelengths  in  the  VIS,  NIR, 
SWIR,  can  be  then  collected. 

For  this  thesis,  signatures  were  obtained  from  the  same  individual  by  getting 
measurements  from  the  dismount’s  skin,  sweatshirt,  jeans,  shirt,  hair  and  the  background 
objects  as  identified  in  Section  4.1.  Figure  4.2  shows  the  location  points  where  the 
signatures  were  taken  using  the  spectroradiometer. 

Per  the  data  sheet  of  the  ASD  FieldSpec  3,  the  spectroradiometer  offers  a  spectral 
range  from  350  nm  to  2500  nm  with  a  spectral  resolution  of  3nm  at  700nm  and  lOnm  at 
1 400/2  lOOnm.  The  amount  of  electromagnetic  energy  received  by  the  spectrometer  at  a 
specific  wavelength  can  then  be  associated  with  the  reflectance  of  a  material  at  that 
wavelength.  Figure  3.6  show  the  various  reflectance  signatures  acquired  with  the 
spectroradiometer  and  used  for  the  Multi-data  dismount  model,  shown  in  Fig.  4.10. 


28 


Figure  3.5:  ASD  FieldSpec  3  Spectroradiometer  used  to  acquire  reflectance  signatures  to 
populate  the  dismount  model  [21]. 


3. 4. 1.1  ASD  Spectroradiometer  Data  Processing.  The  data  acquired  from  the 
ASD  spectroradiometer  has  to  be  post-processed  in  order  to  match  the  data  acquired  with 
the  HST  hyperspectral  camera.  This  is  due  to  the  fact  that  the  resolution  capabilities  and 
the  wavelengths  ranges  covered  by  the  camera  does  not  match  those  acquired  when  using 
the  spectroradiometer,  as  mentioned  in  Section  3.4.2  and  Section  3.4.1  respectively. 

The  process  of  modifying  the  spectrometer  data  is  accomplished  by  data  averaging  to 
match  sampling  interval  and  resolution  of  the  HST.  By  effectuating  this  process  over  all 
the  acquired  spectroradiometer  signatures,  the  data  formatting  is  accomplished.  Now  the 
signatures  from  both  sources  (spectrometer  and  hyperspectral  camera)  can  be  compared. 

This  process  allows  the  evaluation  of  the  feasibility  of  accurately  modeling  a  scene 
using  hyperspectral  data  acquired  by  the  ASD  to  resemble  those  signatures  gathered  with 
a  hyperspectral  camera  of  a  real  scene.  This  discussion  is  documented  in  Section  4.3.  The 
resulting  signatures  after  the  ASD  spectroradiometer  data  adjustment  is  completed  and  the 
data  acquired  with  the  HST  hyperspectral  camera  is  shown  in  Fig.  4.5  trough  4.9. 
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Figure  3.6:  Dismount’s  Hyperspectral  Signatures  from  ASD  FieldSpec  3  Spectroradiome- 
ter.  These  reflectance  signatures  are  used  to  populate  the  dismount  model. 

3.4.2  Signatures  from  Hyperspectral  Camera.  The  hyperspectral  image  is 
generated  by  HyperSpecTIR  (FIST)  hyperspectral  camera.  After  capturing  the  image,  the 
data  is  processed  according  to  the  post-processing  specifications  of  the  device.  Signature 
collection  location  points  are  identified  in  Fig.  4.2  (similar  location  for  all  collection 
sources). 

The  HyperSpecTIR  camera  HST  shown  in  Fig.  3.7,  is  comprised  of  a  pair  of  bore 
sighted  grating  spectrometers  each  operating  in  a  portion  of  the  electro-optic  (EO) 
spectrum  between  450  nm  and  2450  nm.  The  visible  and  near-infrared  (VNIR) 
spectrometer  assembly  operates  over  the  range  of  450  nm  to  900  nm.  The  short  wave 
infrared  (SWIR)  spectrometer  assembly  operates  over  the  range  of  900  nm  to  2450  nm. 
These  two  spectrometers  are  oriented  such  that  they  view  the  scene  through  a  series  of 
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optical  quality  mirrors.  The  instantaneous  field-of-view  (IFOV)  of  the  instrument  is  a  line 
image  of  1  mR  by  256  mR  oriented  along  the  along-track  axis. 


Figure  3.7:  HyperS pecTIR  hyperspectral  camera  [35]. 


Each  collection  is  256  pixels  in  the  y-direction  (along  track)  by  a  variable  number  of 
pixels  in  the  x-direction  (cross-track),  dependent  on  desired  resolution  (lm  res.  =  150 
pixels/  3m  res.  =  550  pixels).  Each  data  cube  produced  by  the  system  covers  the 
wavelengths  between  450  -  2450  nm,  with  about  248  selectable  bands;  240  of  which  are 
data  imagery  bands  and  8  of  which  are  allocated  to  GPS  footer  data  which  is  later  used  in 
post-processing  and  data  verification.  This  GPS  data  is  disregarded  in  the  image  analysis 
for  this  thesis.  Of  these  240  data  imagery  bands,  several  are  redundant  as  a  result  of  the 
spectral  overlap  between  the  two  spectrometers  and  are  removed  during  the 
post-processing  [35]. 

3. 4.2.1  HST  Data  Processing.  After  the  raw  data  is  collected,  further  data 
processing  is  required.  The  post-processing  programs  included  in  the  HST  system  take 
care  of  this  step  [35].  When  the  processing  is  completed,  data  cubes  are  created  and 
hyperspectral  signatures  ( in  radiance  units)  are  obtained  from  the  scene. 
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In  order  to  translate  the  HST  hyperspectral  data  radiance  to  reflectance  values  for 
model  population  and  multi-data  signature  comparison  purposes  additional 
post-processing  is  required.  The  data  output  from  the  HST  hyperspectral  imager  is  in 
radiance,  which  is  the  total  electromagnetic  energy  reflected  from  the  target  which 
includes  environmental  effects  (atmospherical,  green  house,  etc.)  and  is  dependent  on  the 
intensity  of  the  illumination  source.  Empirical  line  correction  (ELC),  is  used  to  find  the 
corresponding  reflectance  for  each  pixel  in  the  image  and  is  calculated  based  on  the  white 
and  gray  calibration  panels  (identified  in  fig.  3.8). 


To  calculate  the  empirical  line  correction  the  following  inputs  are  required: 

•  location  of  the  reference  panels  (Fig.  3.8), 

•  the  hyperspectral  signatures  of  the  panels  from  the  HST  hyperspectral  image  (Fig. 
3.9)  and, 

•  the  hyperspectral  signature  specifications  from  the  manufacturer  for  both  calibration 
boards  (Fig.  3.10). 
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Once  they  are  obtained,  the  determination  of  the  reflectance  signatures  from  the  HST 
radiance  data  can  be  found  for  each  pixel. 


Figure  3.9:  White  (blue  line)  and  gray  (red  line)  boards  radiance  from  the  HST 
hyperspectral  camera  before  ELC  processing. 


The  corrected  signatures  of  the  white  and  gray  boards  are  shown  in  Fig.  3.1 1.  It  can 
be  noticed  that  these  match  those  given  by  the  manufacturer  (Fig.  3.10).  The 
hyperspectral  image  in  its  entirety  can  be  processed  using  the  developed  EEC.  Now 
reflectance  comparison  between  the  Multi-data  signatures  can  take  place.  In  summary,  the 
use  of  Empirical  Line  Correction  and  the  reference  true  reflectance  measurements 
(manufacturer)  from  the  white  and  gray  boards  (Fig.  3.10),  are  necessary  to  determine  the 
corrected  reflectance  measurements.  Hence,  for  further  enhancements  to  the  thesis  it  is 
necessary  that  the  two  reference  reflectance  boards  are  included  in  the  pictures. 
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Figure  3.10:  Reference  reflectance  specification  from  the  manufacturer.  These  signatures 
represent  the  true  wavelengths  responses.  The  blue  line  corresponds  to  the  white  panel 
reflectance  and  the  gray  panel  is  represented  the  red  line. 


3.5  End-Member  Pixel  Mixing 

In  order  to  conservatively  represent  current  hyperspectral  image  acquisition 
capabilities,  end-member  pixel  mixing  is  incorporated  in  the  model.  This  is  accomplished 
by  the  use  of  linear  mixture  model  (LMM).  One  of  the  effects  of  LMM  is  that  it  introduces 
variation  and  certain  level  of  ambiguity  to  the  multi-data  model.  This  ambiguity  will  later 
make  its  effect  in  material  classification.  To  mix  the  end-members  the  edges  of  the 
dismount  and  background  elements  need  to  be  identified.  Section  3.5.1  provides  a  brief 
description  of  the  end  member  finding  method  used  in  this  thesis.  Once  the  edges  are 
located,  LMM  (Section  2.4.1)  is  applied  to  all  of  the  end-members  elements  at  those 
locations  making  an  element  pixel  mixing  based  on  user  defined  class  proportions. 
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Figure  3.11:  The  resulting  hyperspectral  reflectance  measurements  after  the  image  is 
processed  using  ELC  for  the  white  panel  (blue  line)  and  gray  (red  line)  panel. 


3.5.1  Edge  Detector  Module.  There  are  several  ways  to  find  an  edge  in  an  image. 
This  section  provides  a  brief  insight  and  description  of  the  edge  detection  approach  used 
in  this  thesis.  The  problem  of  edge  detection  is  of  great  importance  in  image  analysis.  As 
mentioned  in  Section  2. 1.1.2,  edges  characterize  a  dismount  outline  and  are  therefore 
useful  for  segmentation  and  identification  of  dismounts  in  a  scene. 

An  edge  in  an  image  is  a  jump  in  intensity  [13].  The  cross  section  of  an  edge  has  the 
shape  of  a  ramp.  An  ideal  edge  is  a  discontinuity  (i.e.  a  ramp  with  an  infinite  slope). 
Many  researchers  [27,  40]  have  used  an  error  function  or  smoothed  step  as  the  simplest 
form  of  an  ideal  step  edge  model  for  modeling  the  effects  of  blur  and  shading  in  practical 
application. 
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The  localization  of  edges  is  used  to  identify  end-members  of  an  element  in  the 
model.  For  example,  in  Fig.  3.2,  it  is  desired  to  mix  the  pixels  that  are  at  the  boundary 
(end-members)  of  the  dismount’s  head  (coded  as  1)  and  the  background  (coded  as  0)  to 
recreate  a  mixed  pixel  between  those  materials  and  mimic  current  imagery  technology. 
The  edge  detector  algorithm  (Fig.  3.12)  is  able  to  identify  this  end-members  and  then 
apply  LMM  (Section  3.5)  to  the  model.  An  example  of  the  procedure  is  described  below, 


function  End  Member  Mixing  Algorithm  (Pixel  Matrix,  Hyperspectral 

Signatures,  LMM  Percentages) 
returns:  (Pixel  Mixed  Multi-data  Model) 
inputs:  Pixel  Matrix,  ( m  x  n)  multi -data  model  matrix 

Hyperspectral  Signatures ,  classes  signatures  to  be  mixed 
LMM  percentages ,  mixing  amount  per  class  (user  defined) 
for  l:Size(Pixel  Matrix) 

Identify  end-class-members  pixels 

Determine  end-members  classes  (2  or  more)  in  all  directions 
(up,  down,  left,  right). 

Mix  end-members  classes  using  LMM  (Section  2.1) 
with  user  defined  percentages. 

end  for 

Figure  3.12:  Edge  Detector  Algorithm  with  LMM  implementation. 


Once  the  signatures  are  attached  and  end-member  pixels  are  mixed  the  dismount 
model  is  complete.  If  additional  details,  dismounts  or  background  elements  need  to  be 
added,  the  process  needs  to  go  back  to  the  pixel  map  modification  stage.  The  pixel  map 
will  have  to  be  adjusted  to  reflect  new  changes,  signatures  are  added  and  reattached,  and 
pixels  are  mixed.  This  will  incorporate  any  new  changes  to  the  model  if  needed. 

3.6  Summary 

This  chapter  described  the  various  components  and  algorithms  for  the  creation  of  the 
Multi-data  dismount  model.  In  the  next  chapter  specific  details  of  the  implemented  model, 
the  specifics  about  the  hyperspectral  signatures  applied  and  algorithmic  parameters  are 
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provided.  A  basic  comparison  of  the  hyperspectral  signature  reflectance  measurements 
(ASD  and  HST)  for  model  population  is  conducted  and  documented.  Also,  the  results 
obtained  and  dismount  models  are  presented. 
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4  Experimental  Results  and  Analysis 


This  chapter  documents  the  dismount  and  scene  modeling  process,  obtained  results 
and  their  analysis.  The  developed  multi-data  dismount  model  is  presented  side-by-side 
with  the  hyperspectral  images  signatures  for  comparison.  As  part  of  the  analysis,  a 
straight  forward  comparison,  in  reflectance  terms,  between  modeled  data  and 
hyperspectral  imagery  is  also  conducted.  At  the  same  time,  the  chapter  describes  the 
processes  for  attachment  of  hyperspectral  signatures  from  two  different  sources;  the  first 
being  hyperspectral  signatures  captured  with  HST  camera,  and  the  second,  signatures 
acquired  with  a  Spectroradiometer  (ASD). 

The  chapter  begins  by  discussing  the  acquisition  and  nature  of  the  hyperspectral 
signatures  used  in  this  thesis.  Then  it  justifies  the  use  of  certain  algorithms  and  describes 
specifics  about  the  pixel  map  development  process,  signature  attachment  and  the  created 
multi-data  model.  The  last  section  of  this  chapter  presents  an  analysis  of  the  results  to 
include  an  hyperspectral  dismount  model  comparison  between:  1)  model  populated  with 
signatures  acquired  with  the  spectrometer  [21],  2)  and  an  image  acquired  with  the 
hyperspectral  camera  [35].  Also,  throughout  this  chapter;  advantages,  disadvantages  and 
possible  uses  for  the  computer  aided  multi-data  dismount  model  are  documented. 

4.1  Data  and  Signature  Acquisition 

In  order  to  achieve  the  research  goals  discussed  earlier,  several  hyperspectral  data 
collections  were  necessary.  This  hyperspectral  data  came  from  several  dismount  images 
taken  with  two  hyperspectral  cameras  the  HyperSpecTIR  hyperspectral  [35]  and  PHIRST 
Light  II  cameras  [17].  For  the  case  of  the  HST  hyperspectral  images,  they  were  acquired 
in  a  sunny  day  with  an  environment  temperature  of  about  38  degrees  Fahrenheit  and 
natural  lighting  (Sun),  shown  in  Fig.  4.1.  The  PHIRST  Light  II  collection  took  place  in  an 
indoor  setting  (artificial  lighting)  with  a  temperature  of  about  70  degrees  Fahrenheit. 
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Reflectance  measurements  used  to  populate  the  dismount  model  were  acquired  with 


the  Analytical  Spectral  Devices  (ASD)  FieldSpec  3  Spectroradiometer  [21]. 


Figure  4.1:  Image  captured  with  HST  hyperspectral  camera,  taken  in  the  open  (Sun),  semi- 
urban  environment  in  a  sunny  day. 


To  populate  the  dismount  models,  signature  acquisition  points  are  identified  from 
each  source  (spectrometer  and  hyperspectral  cameras).  Figure  4.2  shows  these  points. 


Figure  4.2:  Location  of  signature  acquisition  points  (yellow  squares)  for  the  two  signature 
acquisition  methods  (spectroradiometer  and  hyperspectral  camera). 
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The  hyperspectral  signatures  varies  depending  on  reflectance  characteristics,  nature 
of  the  material  and  source  of  illumination.  There  is  a  need  that  all  of  the  points  have 
similar  reference  location  and  units  of  measurement  for  multi-data  comparison.  The 
signature  elements  for  the  population  of  the  dismount  model  are  the  following, 

•  background  1  (cork  board) 

•  background  2  (white  calibration  board) 

•  dismount’s  hair 

•  dismount’s  sweatshirt 

•  dismount’s  pants 

•  dismount’s  shirt 

•  dismount’s  skin  (face/cheek) 

The  fact  that  the  location  of  the  signature  points  are  similar  (same  element),  awards 
the  ability  to  compare  multi-data  models  with  similar  locations  from  the  various  sources 
(Section  4.3).  As  well  as  to  compare  the  representation  ability  of  a  “natural”  scene  versus 
the  modeled  scene  (Section  4.3.1). 

4.2  Results  and  Model  Development 

The  developed  model  for  representation  of  the  dismount  and  background  in  a  scene, 
imitates  the  image  taken  with  the  hyperspectral  camera.  The  main  reason  behind  this 
scheme  is  to  be  able  to  conduct  a  basic  study  of  the  ability  of  a  computer  aided  multi-data 
model  to  represent  a  dismount’s  morphological  form  as  well  as  ability  to  represent  a 
“natural/real”  situation  or  scene  acquired  with  a  hyperspectral  camera  (Section  4.3.1). 

The  model’s  pixel  map  is  developed  from  the  images  seen  in  Fig.  4.1  for  HST  and 
Fig.  4.2  for  PHIRST  Light  II  cameras.  This  is  accomplished  using  an  algorithmic 
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modeling  generation  (AMG)  as  explained  in  Section  3.2.  Clustering  and  grouping  the 
various  elements  (Section  4.1)  using  morphological  body  outline  and  body  composition 
by  parts,  give  us  a  basic  template  for  the  pixel  map.  The  pixel  map  or  scene  matrix  that 
resulted  from  this  approach  is  shown  in  Fig.  4.3  for  the  HST  model  and  Fig.  4.4  for  the 
PHIRST  Light  II  model. 
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2  2 
2  1 


Figure  4.3:  Pixel  map  to  mimic  image  taken  with  HST  hyperspectral  camera.  The  pixel 
map  is  the  result  of  clustering  the  scene  image  following  AMG  and  morphological  body 
outline  and  body  composition  by  parts. 


After  the  scene  matrix  is  completed  values  are  assigned  to  the  pixel  map  and  the  pixel 
map’s  legend  is  created  (i.e.  Section  3.2).  For  example,  all  of  the  skin  sections  were 
grouped  as  a  skin  and  given  a  value  of  1  in  the  pixel  map.  The  pixel  maps  legend  for  both 
models  is  shown  in  Table  4.1. 

The  next  step  according  to  the  system  architecture  (Fig.  3.1)  is  the  incorporation  of 
signatures.  For  this  thesis  the  signature  repository  consist  of  hyperspectral  signatures 
acquired  from  the  ASD  Spectroradiometer  (Section  3.4. 1.1). 
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Figure  4.4:  Pixel  map  to  mimic  image  taken  with  PHIRST  Light  II  camera.  The  pixel  map 
is  the  result  of  clustering  the  scene  image  following  AMG  and  morphological  body  outline 
and  body  composition  by  parts. 


Table  4.1:  Pixel  Map  legend  codes  for  Fig.  4.3  and  Fig.  4.4. 


Elements 

Maps  to: 

0 

Background 

1 

Skin 

2 

Hair 

3 

Shirt 

4 

Sweatshirt 

4.2.1  Hyperspectral  Model.  The  hyperspectral  signatures  acquired  with  the  ASD 
Spectroradiometer  can  now  be  processed  and  attached  to  the  pixel  map  according  to  the 
pixel  map’s  legend.  Figures  4.5  trough  4.9  shows  the  resulting  model  from  the  attachment 
of  the  spectrometer  signatures  to  the  scene  model  (Fig.  4.3).  The  figures  include  the 
signature  plots  from  the  HST  hyperspectral  camera  (left)  and  the  Spectroradiometer 
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reflectance  signatures  attached  to  each  element  in  the  model  (right)  with  their  respective 
location. 


Figure  4.5:  Cork  board  model,  populated  using  spectrometer  data  in  the  right  (after 
wavelength  matching,  Section  3.4. 1.1)  vs.  hyperspectral  imagery  in  the  left  (after  HST 
data  processing,  Section  3.4.2. 1)  and  their  respective  attached  reflectance  signatures. 


4.2.2  Signature  Attachment  using  Imagery  Data.  A  second  alternative  for  the 
database  used  to  populate  the  computer  aided  multi-data  model,  is  to  use  signatures 
acquired  from  a  imager.  Signatures  can  be  acquired  from  an  image  by  selecting  a  specific 
section  or  location  where  the  desired  element  is  located  and  populating  the  model. 

One  of  the  disadvantages  of  hyperspectral  sensors  is  the  need  to  incorporate 
atmospheric  correction  algorithms  [11].  Atmospheric  effects  need  to  be  taken  into 
consideration  when  signatures  are  attached  to  the  model  using  signature  directly  captured 
form  imagery  data.  The  nature  of  the  hyperspectral  signatures  will  affect  the  results  of  the 
model.  The  use  of  this  type  of  signatures  present  a  concern,  where  environmental 
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Figure  4.6:  Cheek  (skin)  model,  populated  using  spectrometer  data  in  the  right  (after 
wavelength  matching,  Section  3.4. 1.1)  vs.  hyperspectral  imagery  in  the  left  (after  HST 
data  processing,  Section  3.4.2. 1)  and  their  respective  attached  reflectance  signatures. 


Figure  4.7:  Forehead  (skin)  model,  populated  using  spectrometer  data  in  the  right  (after 
wavelength  matching,  Section  3.4. 1.1)  vs.  hyperspectral  imagery  in  the  left  (after  HST 
data  processing,  Section  3.4.2. 1)  and  their  respective  attached  reflectance  signatures. 
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Figure  4.8:  Sweatshirt  model,  populated  using  spectrometer  data  in  the  right  (after 
wavelength  matching,  Section  3.4. 1.1)  vs.  hyperspectral  imagery  in  the  left  (after  HST 
data  processing,  Section  3.4.2. 1)  and  their  respective  attached  reflectance  signatures. 


Figure  4.9:  White  board  model,  populated  using  spectrometer  data  in  the  right  (after 
wavelength  matching,  Section  3.4. 1.1)  vs.  hyperspectral  imagery  in  the  left  (after  HST 
data  processing,  Section  3.4.2. 1)  and  their  respective  attached  reflectance  signatures. 
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phenomena  (i.e.  atmospheric  effects,  temperature,  humidity  and  others)  are  existent 
during  the  image  collection  event.  These  effects  are  certainly  embedded  in  the 
hyperspectral  signature  and  must  be  taken  into  consideration  if  environmental  effects  play 
a  role  for  employment  and  use  of  the  created  model.  A  better  option  will  be  to  model  the 
environmental  effects  into  the  acquired  data  from  the  Spectroradiometer  and  then  populate 
the  model. 

4.3  Analysis 

This  section  studies  the  ability  of  a  computer  aided  multi-data  model  to  recreate  a 
scene  where  a  dismount  is  present.  It  accomplishes  this  by  comparing  the  created 
hyperspectral  dismount  model  data  versus  data  taken  from  a  hyperspectral  camera  image. 
The  evaluation  and  comparison  of  the  difference  in  reflectance  between  the  two 
approaches  is,  as  well,  documented  in  this  section. 

This  comparison  takes  place  in  order  to  verify  the  feasibility  of  the  model  to  resemble 
images  taken  with  a  hyperspectral  camera.  Additional  analysis  is  necessary  to  conclude 
with  exactitude  the  nature  of  the  different  effects  that  affect  the  direct  resemblance  of  the 
model  and  hyperspectral  HST  camera  data,  and  the  accountability  of  the  same.  Even 
though,  a  hint  of  these  possible  difference  effects  is  given  during  the  analysis  for  future 
consideration. 

At  the  same  time,  supplementary  measurements  like:  spatial  resolution,  atmospheric 
effects,  illumination,  shadowing,  are  examined  for  the  model’s  signatures,  as  well  as  for 
the  image  acquired  using  the  hyperspectral  camera.  Finally,  throughout  this  section  some 
of  the  advantages  and  disadvantages  of  having  a  computer  aided  multi-data  model  versus 
a  hyperspectral  imaging  system  are  mentioned. 

4.3.1  Resemblance  of  Model  vs.  Hyperspectral  Image.  For  the  purpose  of  this 
thesis  the  created  model  mimics  the  image  taken  with  the  hyperspectral  camera.  The 
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objective  of  this  approach  is  to  demonstrate  the  ability  of  the  model  to  represent  a 
hyperspectral  image  utilizing  a  pixel  map.  Figures  4.10  and  4.11  shows  two  hyperspectral 
images  taken  with  a  hyperspectral  cameras  HST  (RGB)  and  PHIRST  Light  II  Camera  and 
their  respective  modeled  scenes  (RGB)  side  by  side. 


Figure  4.10:  Model  emulating  hyper- spectral  image  taken  with  HST  Camera  [35]. 


Figure  4.11:  Model  emulating  hyper-spectral  image  take  from  PHIRST  Light  II  Camera 
(BAE  Systems)  [17]. 


It  is  observable  that  both  models  as  well  as  the  modeled  scenes  (Fig.  4.10  and  Fig. 
4. 1 1  respectively),  have  similar  dismount  outline  and  contain  the  same  background 
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elements.  The  signatures  attached  to  the  model  were  of  RGB  nature  and  was  done  to 
demonstrate  the  ability  of  variation  of  such,  as  seen  in  Fig.  4.11  right  side. 

When  the  models  are  reviewed  against  the  hyperspectral  images,  it  is  also  observable 
that  details  are  lost  in  Figs.  4.10  right  side.  These  loss  of  details  may  be  of  importance  for 
the  purposes  of  dismount  identification.  For  this  occasion,  the  modeled  scene  will  have  to 
be  enhanced  to  comply  with  the  requirements.  It  is  also  noticeable  that  the  “real”  image 
from  the  hyperspectral  camera  HST  (Fig.  4.10  left)  is  out  of  focus,  which  also  may  present 
a  concern  for  dismount  identification  purposes.  An  alternative  solution  to  these  intricacies 
is  to  use  regular  (RGB)  cameras  for  the  objective  of  dismount  identification  and  similar 
applications.  For  the  study  of  dismount  characterization  and  dismount  detection,  the 
dismount  model  is  of  great  advantage  when  compared  to  hyperspectral  camera.  The 
computer  aided  multi-data  model  allows  rapid  modification  of  scenes,  change  of 
signatures,  enhancement,  dismissal  or  modification  of  environmental  effects,  replication 
of  images,  model  reuse  and  many  others. 

Another  observable  difference  is  that  in  the  modeled  scene  there  are  no  shadows 
present.  At  the  present  time,  this  capability  was  not  implemented  in  the  dismount  model, 
but  shadows  can  be  simulated  and  incorporated  on  the  model  in  the  next  developmental 
spiral.  Other  noticeable  differentiation  is  that  illumination  changes  are  not  captured  by  the 
model.  The  changes  in  illumination  could  be  modeled  by  adjusting  the  amplitude  of  the 
attached  signature  where  the  illumination  is  different  (i.e.  specularity).  Again  these 
changes  to  the  model  can  be  added  in  the  next  evolution  of  the  model. 

Every  illumination  source  has  its  own  characteristic  signature.  The  nature  of  the 
illumination  source  or  source  characterization  signatures,  can  be  modeled  and 
implemented  in  the  model  to  obtain  radiance  signatures.  This  could  be  done  after  the 
signature  acquisition  (reflectance)  from  ASD  is  obtained.  The  reflectance  signatures  could 
be  adjusted  to  represent  different  illumination  sources  as  needed  by  the  user.  All  of  these 
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enhancements  can  be  implemented  in  the  near  future  if  required  by  the  user  and 
applications. 

4.3.2  Signal  Comparison  Image  vs.  Modeled  Scene.  The  model  selected  to 
conduct  the  signature  analysis  is  that  from  the  HST  hyperspectral  camera  as  seen  in  Fig. 
4.10.  In  order  to  have  a  basic  comparison  of  the  model,  a  straight  forward  hyperspectral 
signatures  similarity  breakdown  between  model  data  and  HST  (imager)  data  was 
executed.  This  comparison  is  conducted  by  selecting  the  modeled  elements  signatures  and 
imaged  signatures  from  the  hyperspectral  imager.  Both  signatures  are  compared  in 
reflectance  terms  excluding  water  absorption  bands  (shadowed  in  light  blue)  for  difference 
percentage  calculations.  This  percentage  of  difference  reflects  a  measurement  of 
dissimilitude  between  each  other.  Not  to  be  exchanged  with  signal  error. 

For  this  comparison  analysis,  both  signals  are  plotted  and  displayed  conjunctly  in  the 
same  graph.  The  model’s  signature  is  collected  from  the  ASD  spectroradiometer  (red 
dashed  line)  and  the  imager  signature  is  taken  with  the  HST  hyperspectral  imager  (blue 
line).  The  respective  signal  post-processing  adjustments  for  both  ASD  and  HST  are 
carried  out  according  to  Sections  3.4. 1.1  and  3.4.2. 1  respectively.  Difference  measurement 
are  calculated  at  each  wavelength  band  and  averaged  over  the  entire  wavelength  range 
(400  nm  to  2500  nm)  to  obtain  the  percentage  of  difference  for  each  modeled  element  (i.e. 
sweatshirt,  skin,  etc.).  For  this  calculations  environmental  effects  (atmospherical,  green 
house,  water  and  C02  absorption)  around  1400  nm  and  1900  nm  wavelength  bands  were 
dismissed  for  observations  and  conclusion  purposes. 

As  observed  in  Fig.  4.12,  the  difference  between  the  collected  hyperspectral 
signature  from  HST  (blue  line)  and  the  modeled  hyperspectral  signature  signal  (red 
dashed  line)  for  the  white  board  is  minimum.  The  calculated  difference  between  the 
signals  is  1.25%.  The  most  noticeable  difference  between  the  two  signatures  take  place 


49 


Figure  4.12:  Modeled  signature  (red  dashed  line)  and  actual  imager  output  (blue  line)  for 
the  white  board ,  with  water  absorption  frequencies  shadowed  in  light  blue. 


after  the  2100  nm  wavelength  band.  This  could  be  attributed  to  internal  parameters  in  the 
machines  or  environmental  effects. 

For  the  case  of  the  cork  board  signatures,  the  calculated  difference  for  the  selected 
pixels  (Fig.  4.13)  is  2.4%.  It  can  be  noticed  that  the  HST  data  (blue  line)  has  more 
variation  and  is  noisier  than  the  model’s  signature  for  this  background  element.  This  could 
be  attributed  to  atmospheric  effects  as  well  as  imager  effects.  The  resemblance  of  the  two 
signals  is  evident  and  they  have  the  same  shape. 

For  the  white  shirt  area,  the  signals  are  similar  and  follow  the  same  outline.  The 
calculated  difference  percentage  is  1 1.4%.  This  apparently  higher  percentage  could  be  an 
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Figure  4.13:  Model’s  hyperspectral  signature  (red  dashed  line)  and  actual  imager  output 
(blue  line)  for  the  cork  board ,  with  water  absorption  frequencies  shadowed  in  light  blue. 


effect  that  the  measurement  taken  with  the  Spectroradiometer  was  acquired  with  the  shirt 
on  the  dismount  body  and  skin/heat  properties  may  have  influenced  the  final  result. 
Additional  research  may  infer  the  specifics  about  this  effect. 

There  is  a  difference  of  7.67%  between  the  Model  and  the  HST  signatures  for  the 
sweatshirt  pixels.  The  majority  of  this  difference  takes  place  after  900  nm  wavelength. 
Some  of  the  causes  for  this  might  be  because  environmental  temperature  at  the  time  of  the 
capture  was  colder  (38  degrees)  than  that  of  the  ASD  (70  degrees)  or  it  could  also  be 
imager  specific  effects. 
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Figure  4.14:  Model’s  hyperspectral  signature  (red  dashed  line)  and  collected  hyperspectral 
signature  (blue  line)  for  the  shirt,  with  water  absorption  frequencies  shadowed  in  light  blue. 


The  signature  of  the  model  for  the  cheek  area  is  3.32%  different  than  that  of  the 
collected  image.  Although,  both  follow  similar  signal  outline.  It  is  noticeable  that  the  blue 
signal  (HST)  changes  drastically  after  900nm,  which  can  be  referred  to  as  a  imager  effects. 
This  imager  effects  are  noticed  in  all  the  signals  taken  with  the  HST  hyperspectral  imager. 
This  might  be  caused  because  the  HST  hyperspectral  imager  possess  two  spectrometers 
(Section  3.4.2. 1).  At  900  nm  the  imager  moves  to  the  second  spectrometer  and  the  gain  or 
amplitudes  might  be  different  between  the  two.  The  signature  up  to  this  point  of  900  nm  is 
smooth  (minimum  environmental  effects)  and  closely  resemble  that  of  the  model.  After 
this  point  it  becomes  noisier,  but  still  follows  the  general  form  of  that  of  the  model. 
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Figure  4.15:  Model’s  hyperspectral  signature  (red  dashed  line)  and  collected  hyperspectral 
signature  (blue  line)  for  the  Sweatshirt,  with  water  absorption  frequencies  shadowed  in 
light  blue. 


In  the  case  of  the  forehead  area,  which  has  similar  shape  of  that  of  the  cheek  area 
(skin),  both  signatures  (HST  and  model)  resemblance  each  other  and  follow  the  same 
signature  shape.  The  difference  between  them  is  about  2.54%.  Again,  the  imager  effects 
are  also  noticed  around  900  nm  of  wavelength.  The  HST  signature  becomes  unstable  after 
this  measurement,  but  it  still  follows  the  general  characteristics  of  the  model  and 
viceversa.  This  instability  or  noise  may  be  attributed  to  atmospheric  effects,  imager  focus 
and  the  previously  mentioned  imager  effects. 

4.3.3  Signal  Comparison  Mixed  Pixels  in  the  Image  vs.  LMM  Pixel  Mixing 
Modeled  Scene.  After  finding  edge  element  members,  pixel  mixing  is  possible.  Linear 
Mixture  Model  was  the  approach  used  for  pixel  mixing  in  this  thesis  (Section  2.4.1).  In 
regards  to  signature  comparison  (mixed  pixels  model  vs.  mixed  pixels  collected),  edge 
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Figure  4.16:  Model’s  hyperspectral  signature  (red  dashed  line)  and  collected  hyperspectral 
signature  (blue  line)  for  the  Cheek  (skin),  with  water  absorption  frequencies  shadowed  in 
light  blue. 


member  pixels  or  mixed  pixels  in  the  hyperspectral  image  (from  the  HST)  where  visually 
identified  and  used  to  verify  the  calculated  pixel  mixing  performance.  For  example,  in  Fig. 
4.18  a  mixed  pixel  in  the  original  hyperspectral  image  (HST)  is  identified  at  the  skin  and 
cork  board  boundary  region.  This  pixel  contains  skin  and  cork  board  at  sub-pixel  levels. 

After  computing  the  LMM  using  50/50  between  cork  board  and  skin  respectively,  the 
resulting  signature  is  shown  in  Fig.  4.19  (red  dashed  line).  Considering  that  the  selected 
“mixed”  pixel  from  the  HST  (blue  line)  is  taken  from  a  visual  inspection  of  the  figure  it  is 
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Figure  4.17:  Model’s  hyperspectral  signature  (red  dashed  line)  and  collected  hyperspectral 
signature  (blue  line)  for  the  Forehead  (skin),  with  water  absorption  frequencies  shadowed 
in  light  blue. 


observable  that  both  follow  the  same  outline.  The  calculated  difference  for  this  case  is 
8.35%  after  elimination  of  absorption  bands  (marked  in  blue). 

In  the  case  of  a  mix  between  sweatshirt  signature  and  cork  signatures,  results  are 
shown  in  Fig.  4.20.  Again,  imager  effects  and  environmental  effects  are  present  (FIST). 
However,  both  signals,  with  a  LMM  mix  of  50%  sweatshirt  and  50%  cork  signature 
modeled  mix  and  HST  (mixed  pixel),  follow  the  apparent  same  shape.  The  calculated 
difference  between  the  two  signals  (model  vs  imager)  is  of  3.82%. 


55 


Figure  4.18:  Mixed  Pixels  between  Skin  and  Cork  Board  (inside  red  square). 


Wavelength  (nm) 

Figure  4.19:  Model’s  hyperspectral  signature  after  LMM  pixel  mixing  of  50%  skin  and 
50%  cork  board  signature  (red  dashed  line)  vs.  collected  hyperspectral  signature  imager 
output  for  a  mixed  pixel  skin  and  cork  board  region  (blue  line),  with  absorption  frequencies 
shadowed  in  light  blue. 
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Figure  4.20:  Model’s  hyperspectral  signature  after  LMM  pixel  mixing  of  50%  sweatshirt 
and  50%  cork  board  signature  (red  dashed  line)  vs.  collected  hyperspectral  signature 
imager  output  for  a  mixed  pixel  sweatshirt  and  cork  board  region  (blue  line),  with 
absorption  frequencies  shadowed  in  light  blue. 


Figure  4.21  shows  the  results  of  a  50%  white  board  reflectance  signature  and  50% 
cork  board  reflectance  signature  mix  (red  dashed  line)  and  the  signature  of  a  mixed  pixel 
collected  with  the  HST  hyperspectral  imager.  The  difference  between  the  signatures  is 
8.60%.  Environmental  and  imager  specific  effects  may  have  caused  this  difference 
between  the  two  signals. 

Finally,  a  pixel  where  various  elements  (skin,  white  shirt,  sweatshirt  and  corkboard) 
are  present  is  identified  in  the  HST  figure  and  modeled  by  using  a  LMM  pixel  mix  with 
25%  skin,  25%  white  shirt,  25%  sweatshirt  and  25%  corkboard.  The  obtained  results  are 
displayed  in  Fig.  4.22.  It  is  observable  that  both  signatures  are  similar  and  the  calculated 
dissimilitude  is  4.9%. 
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Figure  4.21:  Model’s  hyperspectral  signature  after  LMM  pixel  mixing  of  50%  white  board 
and  50%  cork  board  signature  (red  dashed  line)  vs.  collected  hyperspectral  signature 
imager  output  for  a  mixed  pixel  white  board  and  cork  board  region  (blue  line),  with 
absorption  frequencies  shadowed  in  light  blue. 


After  the  comparison  of  difference  between  the  modeled  and  collected  signals,  it  is 
demonstrated  that  LMM  is  a  substantiable  approach  towards  pixel  mixing.  Environmental 
effects,  as  well  as  the  imager  specific  effects  will  need  to  be  modeled  and  studied  in  order 
to  capture  them  in  the  dismount  model  if  complete  resemblance  is  necessary. 

Other  elements  to  be  taken  into  consideration  for  the  case  of  a  multi-data  dismount 
modeling  in  a  scene,  is  the  spatial  resolution.  Spatial  resolution  does  not  make  a 
considerable  effect  on  the  illustration  of  the  image.  The  main  reason  for  this,  is  because 
the  images  were  taken  in  a  close  proximity  to  the  dismount.  In  the  case  there  is  a  aerial 
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Figure  4.22:  Model’s  hyperspectral  signature  after  LMM  pixel  mixing  of  25%  skin,  25% 
white  shirt,  25%  sweatshirt  and  25%  corkboard  signatures  (red  dashed  line)  vs.  collected 
hyperspectral  signature  imager  output  for  a  mixed  pixel  skin,  white  shirt,  sweatshirt  and 
corkboard  (blue  line),  with  absorption  frequencies  shadowed  in  light  blue. 


collection  or  a  collection  where  the  resolution  parameters  of  the  systems  are  compromised, 
then  the  spatial  resolution  as  well  as  the  pixel  mixing  effects  will  be  evident  in  the  model. 

One  important  observation  from  this  signature  comparison  study  is  that  hyperspectral 
imaging  systems  contain  both  quantitative  and  qualitative  information,  not  only  from  the 
imaging  elements  (dismount,  background,  etc.),  but  system  dependent  characteristics  (i.e. 
processing,  data  manipulation  schemes).  The  wide  range  of  possible  hyperspectral 
imaging  products  prevent  using  a  single  or  multiple  metric(s)  to  characterize  system 
performance  in  a  general  manner.  The  difference  between  the  systems  is  the  approach 
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taken  here.  Thus,  the  developed  multi-data  model  suffers  from  hyperspectral  imaging 
system  inherited  effects,  we  called  imager  effects. 
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5  Conclusion  and  Future  Work 


This  chapter  begins  with  a  discussion  of  the  research  goals  and  the  achievements 
accomplished  by  the  development  of  a  multi-data  dismount  model.  Then  a  brief  summary 
of  results  and  analysis  is  presented.  The  final  section  covers  the  potential  for  future  work 
to  improve  the  developed  computer  aided  multi-data  dismount  modeling  scheme. 

5.1  Summary  of  Methods  and  Conclusion 

The  purpose  of  this  thesis  was  to  create  and  demonstrate  the  feasibility  of  a 
systematic  method  to  construct  a  computer  aided  multi-data  fused  dismount  model  for  the 
demanding  studies  of  dismount  detection,  characterization  and  identification.  The 
developed  system  is  built  on  a  systematic  architecture.  This  architecture  allows  the  ability 
to  represent  a  scene  where  a  dismount  is  (or  is  not)  present  in  a  background.  The  model  is 
capable  of  being  adjusted  to  represent  any  scene  required  by  the  researcher.  The  dismount, 
sections  of  the  dismount,  as  well  as  the  background,  can  be  modeled  and  amended  to 
match  the  necessities  of  the  researcher. 

The  main  focus  of  the  system  is  particularly  in  hyperspectral  signatures  to  the 
dismount  model.  Hyperspectral  imaging  systems  are  being  applied  to  a  number  of  areas, 
including  environmental  studies  and  especially  for  defense  purposes.  Hyperspectral 
signatures  uniquely  capture  spatial  and  spectral  information,  which  allows  the  researcher 
to  use  these  characteristics  to  detect  and  characterize  dismounts. 

The  created  dismount  model  is  capable  of  reaching  multi-data  modeling  advantages 
without  the  need  of  acquiring  expensive  imagery  equipment  and  dedicated  software. 
Therefore,  with  the  described  computer  aided  multi-data  dismount  model  the  researcher 
will  be  able  to  attach  signatures  to  any  desired  modeled  scene.  These  signatures  represent 
current  imagery  equipment  by  the  employment  of  LMM  pixel  mixing  and  by  the  further 
addition  of  atmospheric  modeling  and  imager  noise  specific  characteristics.  Even  though. 
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the  last  two  have  not  been  implemented  in  the  current  development,  the  model  is  able  to 
resemble  a  hyperspectral  image  with  minimal  difference  between  the  two  (about  5%),  as 
demonstrated  in  Section  4.3.  All  of  the  information  required  is  contained  within  the 
dismount  model,  or  can  be  furtherer  incorporated  in  the  multi-data  model  if  necessary  (i.e. 
signature  with  atmospheric,  illumination,  imager  noise  effects). 

It  is  clearly  demonstrated  that  the  developed  multi-data  dismount  model  is  capable  of 
representing  any  desirable  scene  situation  by  the  creation  and  employment  of  the  pixel 
map,  multi-data  signature  attachment  and  pixel  mixing.  Furthermore,  the  model  allows 
the  user  to  easily  modify  the  posture,  shape,  location,  size,  position,  multi-data  signatures, 
as  well  as  the  modification  of  the  background  elements  and  characteristics  in  the  scene. 
Multi-data  signatures  (i.e.  hyperspectral,  multispectral,  RGB)  are  attached  to  the  pixel 
map,  and  edge  members  pixels  are  located  and  mixed  to  generate  the  final  modeled  scene. 
By  the  employment  of  LMM  and  pixel  mixing,  the  model  is  able  to  mimic  current 
imaging  technologies  with  a  proven  resemblance  between  hyperspectral  signatures,  mixed 
signatures  and  its  imaged  counterparts. 

After  hyperspectral  signature  analysis  and  model  data  versus  imager  data 
comparison,  it  is  found  that  signatures  acquired  with  the  ASD  Spectroradiometer  (model) 
closely  resemble  those  acquired  from  the  hyperspectral  imager  (Section  4.3.2).  The  model 
shows  notable  distinctions  when  compared  with  a  hyperspectral  image,  these  include  the 
loss  of  details,  loss  of  shadows  and  changes  in  illumination  (between  pixels)  are  lost.  This 
could  be  modeled  in  the  future  and  incorporated  in  the  multi-data  dismount  model  if 
necessary.  Also,  the  model  dismisses  imager  effects  and  environment  effects  if  the 
signatures  are  acquired  with  the  ASD  Spectroradiometer. 

For  the  purposes  of  the  study  of  dismount  characterization  and  dismount  detection, 
the  model  is  of  great  advantage  when  compared  to  hyperspectral  imager.  The  computer 
aided  multi-data  model  allows  rapid  modification  of  scenes,  change  of  signatures, 
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enhancement,  dismissal  or  modification  of  environmental  effects,  replication  of  images, 
model  reuse  and  more.  The  capability  of  creating  a  dismount  model,  where  multi-data 
signatures  are  fused  to  the  model  and  the  ability  of  easy  manipulation  and  development, 
considerably  streamlines  the  research  processes.  The  model  also  grows  the  potential  of 
generating  improvements  on  characterization  of  individuals  by  an  efficient,  easy  to 
acquire,  quick  to  develop,  dismount  modeling  design.  This  comprehensive  modeling 
capability  provides  the  competence  to  explore  hyperspectral  imaging  methods  and  exploit 
hyperspectral  signatures  unique  characteristics  for  dismount  detection  and 
characterization  endeavors. 

5.2  Future  Work 

The  goal  of  creating  a  platform  for  the  study  and  modification  of  dismount  models 
motivated  our  multi-data  dismount  system  modeling.  Despite  the  evidence  of  this  success, 
there  are  still  a  variety  of  avenues  in  which  details  and  enhancements  to  the  model  may  be 
incorporated. 

Certainly,  a  major  area  for  study  will  be  the  modification  of  Spectroradiometer  data 
(model’s  signatures)  to  copy  that  acquired  with  hyperspectral  imagers.  A  sample  data  set 
of  hyperspectral  images  are  required  to  produce  a  model  of  a  variety  of  imager  specific 
noise  effects,  atmospheric  and  environmental  conditions.  These  models  will  enhance  the 
ability  of  the  multi-data  dismount  model  to  reflect  any  atmospherics  and  distinctive 
imager  specific  requirements. 
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